Skip to content

Conversation

@oflatt
Copy link
Contributor

@oflatt oflatt commented Jul 30, 2024

Description of changes

This PR implements RFC #74 behind a new feature flag "entity-manifest".
It adds:

  • A new public function compute_entity_manifest (with experimental warning)
  • New structsEntityManifest, RootAccessTrie, AccessTrie, EntitySliceError. (with experimental warning)
  • JSON serialization for the entity manifest, along with tests that use it.

Note that there's a TODO in the compute_entity_manifest function about returning type errors.

Follow up PRs

  • Add Entity Slicing using Entity Manifests to Cedar #1105 adds an entity slicing function that produces an Entities store from an existing Entities store. This enables strong corpus testing.
  • Generalization to all of Cedar
  • Type annotated entity manifests
  • Support for a SimpleEntityLoader as described in the RFC.
  • Support for an easy to use flattened format for the entity manifest, and a human-readable format.

Checklist for requesting a review

The change in this PR is (choose one, and delete the other options):

  • A breaking change requiring a major version bump to cedar-policy (e.g., changes to the signature of an existing API).
  • A backwards-compatible change requiring a minor version bump to cedar-policy (e.g., addition of a new API).
  • A bug fix or other functionality change requiring a patch to cedar-policy.
  • A change "invisible" to users (e.g., documentation, changes to "internal" crates like cedar-policy-core, cedar-validator, etc.)
  • A change (breaking or otherwise) that only impacts unreleased or experimental code.

I confirm that this PR (choose one, and delete the other options):

  • Updates the "Unreleased" section of the CHANGELOG with a description of my change (required for major/minor version bumps).
  • Does not update the CHANGELOG because my change does not significantly impact released code.

I confirm that cedar-spec (choose one, and delete the other options):

  • Does not require updates because my change does not impact the Cedar formal model or DRT infrastructure.
  • Requires updates, and I have made / will make these updates myself. (Please include in your description a timeline or link to the relevant PR in cedar-spec, and how you have tested that your updates are correct.)
  • Requires updates, but I do not plan to make them in the near future. (Make sure that your changes are hidden behind a feature flag to mark them as experimental.)
  • I'm not sure how my change impacts cedar-spec. (Post your PR anyways, and we'll discuss in the comments.)

@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch from 0920ec4 to 3bc120e Compare July 30, 2024 19:41
@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch 2 times, most recently from c081b68 to 266ba3b Compare July 30, 2024 21:04
@oflatt oflatt changed the title Add Entity Manifests to Cedar (Pending RFC Approval) Add Entity Manifests to Cedar Jul 31, 2024
Ok(RootAccessTrie::new())
}

// TODO is returning the first error correct?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need advice on this TODO

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning the first error is fine, but returning all the errors would be more consistent with the rest of the validation APIs.

Re your second question: yes, validation does some additional steps (e.g., handling for templates). If you look at validate and validate_policies here you'll see that there are several pre-processing steps before we get to typecheck_policy (which is what calls typecheck_by_request_env). I think it'd be ideal to add you changes to the main validation path (if possible).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how to add it to the validation path without it getting ugly- it would have to return both the current result and an optional entity manifest.
I'll keep this as-is for now.

Copy link
Contributor Author

@oflatt oflatt Aug 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ended up running full validation first before this code path, giving a good error message

Copy link
Contributor

@cdisselkoen cdisselkoen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely a reasonable implementation of the RFC as it currently stands. Some low-level comments, but at a high level this looks great.

@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch 2 times, most recently from bace765 to a427a9b Compare August 1, 2024 18:40
Ok(RootAccessTrie::new())
}

// TODO is returning the first error correct?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning the first error is fine, but returning all the errors would be more consistent with the rest of the validation APIs.

Re your second question: yes, validation does some additional steps (e.g., handling for templates). If you look at validate and validate_policies here you'll see that there are several pre-processing steps before we get to typecheck_policy (which is what calls typecheck_by_request_env). I think it'd be ideal to add you changes to the main validation path (if possible).

@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch from b768da6 to 23caaa0 Compare August 26, 2024 22:15
@oflatt oflatt requested review from khieta and shaobo-he-aws August 26, 2024 22:21
@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch from 469d262 to f345d3c Compare August 27, 2024 16:15
Copy link
Contributor

@john-h-kastner-aws john-h-kastner-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Main point of concern in the assumption that validation errors will be non-empty.

#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Hash)]
pub enum EntityRoot {
/// Literal entity ids
Literal(EntityUID),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will end up exporting cedar_policy_core::ast::EntityUID from cedar_policy. Not a blocking issue for an experimental feature, but we'll need to introduce some more wrapper type in cedar_policy if we want callers to be able to use this directly as a cedar_policy::EntityUID. Same comment for Var variant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that this should use the existing wrapper EntityUid?
That's a little unfortunate since I can't import it from the validator crate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, Callers should be able to use it as they would the existing EntityUid wrapper. It does get a annoying since you'll end up wanting a wrapper for most of your structs, but we don't want consumers of cedar-policy to ever import cedar-policy-core

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other option would be using the opaque inner struct pattern we use for error messages if users don't need to look inside the Literal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. I think people should be able to get the literal, so for now I'll add a TODO to implement wrappers for everything in api.rs

}

PolicyCheck::Fail(errors) => {
// PANIC SAFETY: policy check fail should be a non-empty vector.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumption might not hold. Check behavior on undefined action and entity types

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning the full list of errors would avoid any uncertainty here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea
How should I implement Diagnostic for multiple validation errors? Should it print out the first one? What about when there are no errors?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can take a look at ParseErrors in Core for precedent. To the first question, at least in that case we chose to just do the first one. To the second question, we used NonEmpty to structurally guarantee we can't have an empty vector.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to the pointer to ParseErrors
I'm still having trouble finding the case where there are no errors reported- could you point me in the right direction? What are undefined action and entity types?

Copy link
Contributor Author

@oflatt oflatt Aug 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One option could be to

  1. Run full strict mode validation, getting a proper api::ValidationResult
  2. Run typecheck_by_request_env to get the type-annotated policies, using panic if we encounter a type error

Another option would be to embed entity manifest computation into the typechecker to make this more efficient. But this seems kind of ugly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still having trouble finding the case where there are no errors reported- could you point me in the right direction? What are undefined action and entity types?

// `None` if the action entity is not defined in the schema.
// This will only show up if we're typechecking with a
// request environment that was not constructed from the
// schema cross product, which will not happen through our
// public entry points, but it can occur if calling
// `typecheck` directly which happens in our tests.
None => TypecheckAnswer::fail(
ExprBuilder::new().with_same_source_loc(e).var(Var::Action),
),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my case, all of the request environments are from the schema cross product so we should be okay I think.
But either way, I've rewritten the code to do full validation first, then get the type-annotated policies.

@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch 3 times, most recently from 2445b74 to ad8e43b Compare August 27, 2024 20:23
@oflatt oflatt mentioned this pull request Aug 28, 2024
11 tasks
Copy link
Contributor

@khieta khieta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I just left some small nit-picky comments

@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch from 793c7ea to d2f0caa Compare August 29, 2024 20:46
oflatt and others added 14 commits August 29, 2024 14:52
Signed-off-by: oflatt <[email protected]>
This reverts commit 65c1324.

Signed-off-by: oflatt <[email protected]>
Signed-off-by: oflatt <[email protected]>
Signed-off-by: oflatt <[email protected]>
Co-authored-by: Kesha Hietala <[email protected]>
Signed-off-by: oflatt <[email protected]>
Signed-off-by: oflatt <[email protected]>
Signed-off-by: oflatt <[email protected]>
@oflatt oflatt force-pushed the oflatt-entity-manifest-minimal branch from d9b2950 to ad96bd3 Compare August 29, 2024 20:52
@oflatt oflatt mentioned this pull request Aug 30, 2024
11 tasks
@oflatt oflatt merged commit 919a916 into cedar-policy:main Sep 3, 2024
andrewmwells-amazon added a commit that referenced this pull request Sep 4, 2024
andrewmwells-amazon added a commit that referenced this pull request Sep 4, 2024
This reverts commit 919a916.

Signed-off-by: Andrew Wells <[email protected]>
andrewmwells-amazon added a commit that referenced this pull request Sep 4, 2024
shaobo-he-aws added a commit that referenced this pull request Sep 26, 2024
shaobo-he-aws added a commit that referenced this pull request Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants